{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. 当采用带正则的模型以及采用随机梯度下降优化算法时，需要对输入（连续型）特征进行去量纲预处理。课程代码给出了用标准化（StandardScaler）的结果，请改成最小最大缩放（MinMaxScaler）去量纲，并重新训练最小二乘线性回归、岭回归、和Lasso模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      CRIM  ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD  TAX  PTRATIO  \\\n",
      "0  0.00632  18   2.31     0  0.538  6.575  65.2  4.0900    1  296       15   \n",
      "1  0.02731   0   7.07     0  0.469  6.421  78.9  4.9671    2  242       17   \n",
      "2  0.02729   0   7.07     0  0.469  7.185  61.1  4.9671    2  242       17   \n",
      "3  0.03237   0   2.18     0  0.458  6.998  45.8  6.0622    3  222       18   \n",
      "4  0.06905   0   2.18     0  0.458  7.147  54.2  6.0622    3  222       18   \n",
      "\n",
      "        B  LSTAT  MEDV  \n",
      "0  396.90   4.98  24.0  \n",
      "1  396.90   9.14  21.6  \n",
      "2  392.83   4.03  34.7  \n",
      "3  394.63   2.94  33.4  \n",
      "4  396.90   5.33  36.2  \n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 506 entries, 0 to 505\n",
      "Data columns (total 14 columns):\n",
      "CRIM       506 non-null float64\n",
      "ZN         506 non-null int64\n",
      "INDUS      506 non-null float64\n",
      "CHAS       506 non-null int64\n",
      "NOX        506 non-null float64\n",
      "RM         506 non-null float64\n",
      "AGE        506 non-null float64\n",
      "DIS        506 non-null float64\n",
      "RAD        506 non-null int64\n",
      "TAX        506 non-null int64\n",
      "PTRATIO    506 non-null int64\n",
      "B          506 non-null float64\n",
      "LSTAT      506 non-null float64\n",
      "MEDV       506 non-null float64\n",
      "dtypes: float64(9), int64(5)\n",
      "memory usage: 55.4 KB\n",
      "None\n",
      "   RAD_1  RAD_2  RAD_3  RAD_4  RAD_5  RAD_6  RAD_7  RAD_8  RAD_24\n",
      "0      1      0      0      0      0      0      0      0       0\n",
      "1      0      1      0      0      0      0      0      0       0\n",
      "2      0      1      0      0      0      0      0      0       0\n",
      "3      0      0      1      0      0      0      0      0       0\n",
      "4      0      0      1      0      0      0      0      0       0\n",
      "       CRIM    ZN     INDUS  CHAS       NOX        RM       AGE       DIS  \\\n",
      "0  0.000000  0.18  0.067815   0.0  0.314815  0.577505  0.641607  0.269203   \n",
      "1  0.000236  0.00  0.242302   0.0  0.172840  0.547998  0.782698  0.348962   \n",
      "2  0.000236  0.00  0.242302   0.0  0.172840  0.694386  0.599382  0.348962   \n",
      "3  0.000293  0.00  0.063050   0.0  0.150206  0.658555  0.441813  0.448545   \n",
      "4  0.000705  0.00  0.063050   0.0  0.150206  0.687105  0.528321  0.448545   \n",
      "\n",
      "        TAX  PTRATIO  ...  RAD_2  RAD_3  RAD_4  RAD_5  RAD_6  RAD_7  RAD_8  \\\n",
      "0  0.208015      0.3  ...      0      0      0      0      0      0      0   \n",
      "1  0.104962      0.5  ...      1      0      0      0      0      0      0   \n",
      "2  0.104962      0.5  ...      1      0      0      0      0      0      0   \n",
      "3  0.066794      0.6  ...      0      1      0      0      0      0      0   \n",
      "4  0.066794      0.6  ...      0      1      0      0      0      0      0   \n",
      "\n",
      "   RAD_24      MEDV  log_MEDV  \n",
      "0       0  0.422222  0.666856  \n",
      "1       0  0.368889  0.619696  \n",
      "2       0  0.660000  0.833335  \n",
      "3       0  0.631111  0.816001  \n",
      "4       0  0.693333  0.852567  \n",
      "\n",
      "[5 rows x 23 columns]\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 506 entries, 0 to 505\n",
      "Data columns (total 23 columns):\n",
      "CRIM        506 non-null float64\n",
      "ZN          506 non-null float64\n",
      "INDUS       506 non-null float64\n",
      "CHAS        506 non-null float64\n",
      "NOX         506 non-null float64\n",
      "RM          506 non-null float64\n",
      "AGE         506 non-null float64\n",
      "DIS         506 non-null float64\n",
      "TAX         506 non-null float64\n",
      "PTRATIO     506 non-null float64\n",
      "B           506 non-null float64\n",
      "LSTAT       506 non-null float64\n",
      "RAD_1       506 non-null uint8\n",
      "RAD_2       506 non-null uint8\n",
      "RAD_3       506 non-null uint8\n",
      "RAD_4       506 non-null uint8\n",
      "RAD_5       506 non-null uint8\n",
      "RAD_6       506 non-null uint8\n",
      "RAD_7       506 non-null uint8\n",
      "RAD_8       506 non-null uint8\n",
      "RAD_24      506 non-null uint8\n",
      "MEDV        506 non-null float64\n",
      "log_MEDV    506 non-null float64\n",
      "dtypes: float64(14), uint8(9)\n",
      "memory usage: 59.9 KB\n",
      "None\n",
      "The r2 score of LinearRegression on test is 0.743281583749811\n",
      "The r2 score of LinearRegression on train is 0.7470327232427956\n",
      "The r2 score of RidgeCV on test is 0.7435273582381465\n",
      "The r2 score of RidgeCV on train is 0.7469661487604818\n",
      "alpha is: 0.1\n",
      "The r2 score of LassoCV on test is 0.7431014841533542\n",
      "The r2 score of LassoCV on train is 0.7469946254210429\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "E:\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alpha is: 3.5844456970449495e-05\n"
     ]
    }
   ],
   "source": [
    "import numpy as np  # 矩阵操作\n",
    "import pandas as pd # SQL数据处理\n",
    "\n",
    "# path to where the data lies\n",
    "#dpath = './data/'\n",
    "df = pd.read_csv(\"boston_housing.csv\")\n",
    "\n",
    "#通过观察前5行，了解数据每列（特征）的概况\n",
    "print(df.head())\n",
    "\n",
    "#显示数据基本信息\n",
    "print(df.info())\n",
    "\n",
    "# 从原始数据中分离输入特征x和输出y\n",
    "y = df['MEDV']\n",
    "X = df.drop('MEDV', axis = 1)\n",
    "\n",
    "# 尝试对y（房屋价格）做log变换，对log变换后的价格进行估计\n",
    "log_y = np.log1p(y)\n",
    "\n",
    "# RAD的含义是距离高速公路的便利指数。虽然给的数值是数值型，但实际是索引，可换成离散特征/类别型特征编码试试。\n",
    "X[\"RAD\"].astype(\"object\")\n",
    "X_cat = X[\"RAD\"]\n",
    "X_cat = pd.get_dummies(X_cat, prefix=\"RAD\")\n",
    "\n",
    "X = X.drop(\"RAD\", axis = 1)\n",
    "\n",
    "#特征名称，用于保存特征工程结果\n",
    "feat_names = X.columns\n",
    "print(X_cat.head())\n",
    "\n",
    "# 数据标准化\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# 分别初始化对特征和目标值的标准化器\n",
    "ss_X = MinMaxScaler()\n",
    "ss_y = MinMaxScaler()\n",
    "\n",
    "ss_log_y = MinMaxScaler()\n",
    "\n",
    "\n",
    "# 分别对训练和测试数据的特征以及目标值进行标准化处理\n",
    "# 对训练数据，先调用fit方法训练模型，得到模型参数；然后对训练数据和测试数据进行transform\n",
    "from sklearn.linear_model import LinearRegression\n",
    "X = ss_X.fit_transform(X)\n",
    "\n",
    "#对y做标准化不是必须\n",
    "#对y标准化的好处是不同问题的w差异不太大，同时正则参数的范围也有限\n",
    "y = ss_y.fit_transform(y.values.reshape(-1, 1))\n",
    "log_y = ss_y.fit_transform(log_y.values.reshape(-1, 1))\n",
    "\n",
    "fe_data = pd.DataFrame(data = X, columns = feat_names, index = df.index)\n",
    "fe_data = pd.concat([fe_data, X_cat], axis = 1, ignore_index=False)\n",
    "\n",
    "#加#加上标签y\n",
    "fe_data[\"MEDV\"] = y\n",
    "fe_data[\"log_MEDV\"] = log_y\n",
    "\n",
    "#保存结果到文件\n",
    "fe_data.to_csv('FE_boston_housing.csv', index=False)\n",
    "\n",
    "print(fe_data.head())\n",
    "print(fe_data.info())\n",
    "\n",
    "\n",
    "\n",
    "import numpy as np  # 矩阵操作\n",
    "import pandas as pd # SQL数据处理\n",
    "\n",
    "from sklearn.metrics import r2_score  #评价回归预测模型的性能\n",
    "\n",
    "import matplotlib.pyplot as plt   #画图\n",
    "import seaborn as sns\n",
    "\n",
    "# 图形出现在Notebook里而不是新窗口\n",
    "#\n",
    "df = pd.read_csv(\"FE_boston_housing.csv\")\n",
    "# 从原始数据中分离输入特征x和输出y\n",
    "y = df[\"MEDV\"]\n",
    "\n",
    "X = df.drop([\"MEDV\", \"log_MEDV\"], axis = 1)\n",
    "\n",
    "#特征名称，用于后续显示权重系数对应的特征\n",
    "feat_names = X.columns\n",
    "\n",
    "\n",
    "#将数据分割训练数据与测试数据\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# 随机采样20%的数据构建测试样本，其余作为训练样本\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=23, test_size=0.2)\n",
    "X_train.shape\n",
    "#506*0.8,没问题，随机种子不想改！！！\n",
    "#不改貌似作业和参考一样，改了就不一样了。。。真香\n",
    "\n",
    "# 线性回归\n",
    "#class sklearn.linear_model.LinearRegression(fit_intercept=True, normalize=False, copy_X=True, n_jobs=1)\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# 使用默认配置初始化学习器实例\n",
    "lr = LinearRegression()\n",
    "\n",
    "# 用训练数据训练模型参数\n",
    "lr.fit(X_train, y_train)\n",
    "\n",
    "#  用训练好的模型对测试集进行预测\n",
    "y_test_pred_lr = lr.predict(X_test)\n",
    "y_train_pred_lr = lr.predict(X_train)\n",
    "\n",
    "\n",
    "# 看看各特征的权重系数，系数的绝对值大小可视为该特征的重要性\n",
    "fs = pd.DataFrame({\"columns\":list(feat_names), \"coef\":list((lr.coef_.T))})\n",
    "fs.sort_values(by=['coef'],ascending=False)\n",
    "\n",
    "\n",
    "# 使用r2_score评价模型在测试集和训练集上的性能，并输出评估结果\n",
    "#测试集\n",
    "print('The r2 score of LinearRegression on test is',r2_score(y_test, y_test_pred_lr))\n",
    "#训练集\n",
    "print('The r2 score of LinearRegression on train is',r2_score(y_train, y_train_pred_lr))\n",
    "\n",
    "\n",
    "#岭回归／L2正则\n",
    "#class sklearn.linear_model.RidgeCV(alphas=(0.1, 1.0, 10.0), fit_intercept=True,\n",
    "#                                  normalize=False, scoring=None, cv=None, gcv_mode=None,\n",
    "#                                  store_cv_values=False)\n",
    "from sklearn.linear_model import  RidgeCV\n",
    "\n",
    "#设置超参数（正则参数）范围\n",
    "alphas = [ 0.01, 0.1, 1, 10,100]#λ这个东西影响很大\n",
    "#\n",
    "#生成一个RidgeCV实例\n",
    "ridge = RidgeCV(alphas=alphas, store_cv_values=True)\n",
    "\n",
    "# 模型训练\n",
    "ridge.fit(X_train, y_train)\n",
    "\n",
    "#预测\n",
    "y_test_pred_ridge = ridge.predict(X_test)\n",
    "y_train_pred_ridge = ridge.predict(X_train)\n",
    "\n",
    "\n",
    "# 评估，使用r2_score评价模型在测试集和训练集上的性能\n",
    "print('The r2 score of RidgeCV on test is', r2_score(y_test, y_test_pred_ridge))\n",
    "print('The r2 score of RidgeCV on train is', r2_score(y_train, y_train_pred_ridge))\n",
    "\n",
    "print ('alpha is:', ridge.alpha_)\n",
    "\n",
    "# 看看各特征的权重系数，系数的绝对值大小可视为该特征的重要性\n",
    "fs = pd.DataFrame({\"columns\":list(feat_names), \"coef_lr\":list((lr.coef_.T)), \"coef_ridge\":list((ridge.coef_.T))})\n",
    "fs.sort_values(by=['coef_lr'],ascending=False)\n",
    "\n",
    "\n",
    "# Lasso／L1正则\n",
    "# class sklearn.linear_model.LassoCV(eps=0.001, n_alphas=100, alphas=None, fit_intercept=True,\n",
    "#                                    normalize=False, precompute=’auto’, max_iter=1000,\n",
    "#                                    tol=0.0001, copy_X=True, cv=None, verbose=False, n_jobs=1,\n",
    "#                                    positive=False, random_state=None, selection=’cyclic’)\n",
    "from sklearn.linear_model import LassoCV\n",
    "\n",
    "# 生成LassoCV实例（默认超参数搜索范围）\n",
    "lasso = LassoCV()\n",
    "\n",
    "# 训练（内含CV）\n",
    "lasso.fit(X_train, y_train)\n",
    "\n",
    "# 测试\n",
    "y_test_pred_lasso = lasso.predict(X_test)\n",
    "y_train_pred_lasso = lasso.predict(X_train)\n",
    "\n",
    "# 评估，使用r2_score评价模型在测试集和训练集上的性能\n",
    "print('The r2 score of LassoCV on test is', r2_score(y_test, y_test_pred_lasso))\n",
    "print('The r2 score of LassoCV on train is', r2_score(y_train, y_train_pred_lasso))\n",
    "\n",
    "# %%\n",
    "\n",
    "mses = np.mean(lasso.mse_path_, axis=1)\n",
    "plt.plot(np.log10(lasso.alphas_), mses)\n",
    "# plt.plot(np.log10(lasso.alphas_)*np.ones(3), [0.3, 0.4, 1.0])\n",
    "plt.xlabel('log(alpha)')\n",
    "plt.ylabel('mse')\n",
    "plt.show()\n",
    "\n",
    "print('alpha is:', lasso.alpha_)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. 代码中给出了岭回归（RidgeCV）和Lasso（LassoCV）超参数（alpha_）调优的过程，请结合两个最佳模型以及最小二乘线性回归模型的结果，给出什么场合应该用岭回归，什么场合用Lasso，什么场合用最小二乘。\n",
    "答：在什么样场合选择什么样的模型还没有弄懂，请老师在课上详细讲解。我的理解是当输入特征不相关时用最小二乘，当输入特征存在共线性情况，用岭回归可以得定稳定解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
